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ABSTRACT 

gained much support from the educational community, many questions arise as 
specific implementations are attempted. This study examined one aspect, 
namely, the content validity of specific requirements, and addressed the 
question "How do various constituencies (methods students, student teachers, 
teacher educators, and practicing teachers) rate the utility of various 
requirements for student teaching and future teaching in general?" The 
secondary methods instructor at a small teacher education institution 
designed 15 possible outcomes to be met, and student portfolio submissions 
were required for all 15 during the spring 1997 semester. At the conclusion 
of the semester, these various requirements were validated for perceived 
utility by comparing the rank order data collected from the following: (1) 

students just completing methods courses; (2) students just completing their 
student teaching; (3) practicing teachers from the spring 1997 semester; and 
(4) teacher educators. Results indicated that there were no differences in 
rank order correlations, suggesting that all four constituencies agreed on 
the relative importance of the items. These findings have important 
implications for future designers of portfolios. They may trust that the 
perception of utility is equivalent across the various groups. (Contains 4 
tables and 18 references.) (SLD) 
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Abstract 

One advantage presented by the movement toward portfolio assessment is the public 
validation of the learners’ achievements. This aspect is especially important for teacher 
education, as education has been under "heavy fire" by many critics, from within and from 
without the profession. While the value of portfolio learning and assessment has gained much 
support from the educational community, many questions arise as specific implementations are 
attempted. This study examined one aspect, namely, the content validity of specific 
requirements. The question addressed by this study is, "How do various constituencies (methods 
students, student teachers, teacher educators, and practicing teachers) rate the utility of various 
requirements for student teaching and future teaching in general?" 

The secondary methods instructor at a small teacher education institution designed 15 
possible outcomes to be met and student portfolio submissions were required for all 15 during 
the Spring 1997 Semester. At the conclusion of the semester, these various requirements were 
validated for perceived utility by comparing the rank order data collected from the following: 
(1) students just completing the methods course; (2) students just completing their student 
teaching; (3) practicing teachers from the Spring 1997 Semester; and (4) teacher educators. 
Results indicated that there were no differences in rank order correlations, suggesting that all 
four constituencies agreed on the relative importance of the items. These findings have important 
implications for future designers of portfolios; they may trust that the perceptions of utility is 
equivalent across the various groups. 
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Review of Literature 

Portfolios are becoming the assessment of choice for many teacher education 
programs(Barton & Collins, 1993; Stahle & Mitchell, 1993; Ross, 1996; James & VanCleaf, 
1990; Mokhatari, Yellin, Bull, & Montgomery, 1996; Dollase, 1996). Authentic assessment, 
such as portfolios, include complete, integrative tasks which call for the use and application of 
different types of knowledge and skills. These measures are more flexible than traditional 
measures have been, allowing for the consideration of students’ personal learning styles. 

Performance-based measures encourage reflection and analysis by the student (McFarland, 
1994). 

The value of portfolio learning and assessment has gained much support from the 
educational community and seems to be consistent with other changes in educational practice. 
For example, the research on learning supports the aspects of time-on-task and distributed 
practice shown to be important by Dempster (1991); portfolios are consistent with this reform. 
Their use is also consistent with the move to use portfolios in state and national certification 
(Costantino & DeLorenzo, 1994; Kendall, 1994). Portfolios encourage student responsibility 
and student ownership of the learning process in teacher education (Ross, 1996). Portfolios help 
create meaningful "rites of passage" type experiences for methods students and indicate growth 
m student’s abilities. -Their use has helped clarify program goals and course objectives (Barton 
& Collins, 1993). Finally, because portfolios are authentic and dynamic, they are thought to 
better capture the reality of a student’s competence (Krause, 1996). 

Empirical evidence, however, is lacking in research. Herman and Winters (1994) 
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reported that most portfolio articles written were anecdotal or conceptual and few reported actual 
research-based results. 

One reason for the scarcity of data may be that portfolios are in their infancy stage. As 
schools of education define and refine portfolio content and application, assessment policies will 
continuously evolve. Due to the unstandardized nature and content of portfolios, assessment of 
such and the reliability and validity of the assessment become issues. This study attempts to 
provide evidence toward the content validity of the portfolio used at one teacher education 
institution. 

In one previous attempt to establish reliability and validity, Naizer (1997) used a 
traditional measure (short-answer/discussion final examination) to correlate with performance- 
portfolio scores. This study found a very low (Pearson’s correlation = .22) correlation between 
the final examination and performance-portfolio scores. This may have been due to the fact that 
the tests were measuring different information. McFarland (1994) found high correlations 
(Pearson’s correlation = .98 and .96) for vocabulary subtests (standardized tests), while finding 
lower correlations (Pearson’s correlation = .60 to .19) on the various performance subtests in 
the area of science. However, the use of standardized tests to evaluate performance-based tests 
has limitations: test form bias may have understated the "scores" of portfolio based learners. 

Other approaches show scant evidence for the effectiveness of teaching portfolios, yet 
the arguments endure •(Wo If,- Whinery, & Hagerty, 1995). James and Van Cleaf (1990) suggest 
that preservice teacher portfolios should meet the needs of employers, college faculty, and 
preservice teachers. Tucker (1991) suggests that a large part of the evaluation of the student’s 
competence should be project work, much of it done in collaboration with other students. 
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Currently our educational system tends to encourage individual, competitive performance. 
However, one of the chief complaints from the workplace is that the new employees must have 
the capacity to function as members of a team (McFarland, 1994). Portfolios are more likely to 
encourage group interaction and collaboration to enhance the learning process as well as improve 
the quality of portfolio submissions for assessment. 

Finally, the purpose of the portfolio must be clearly defined to the student. The portfolio 
is something that is done by the student, not to the student (Paulson, Paulson, & Meyer, 1991). 
The portfolio becomes a living document which is an exhibit of the student’s work. Portfolios 
must evidence student self-reflection as well as promote ownership over the learning process, 
enhance teaching, and open up assessment. The ultimate purpose of the portfolio is the 
improvement of teacher effectiveness. 

Therefore the contents of the portfolio are of utmost importance. More argumentative 
controversy exists in this area. Simmons (1996) suggests that many portfolios used as evaluation 
systems, are overly prescriptive and more control of contents by the students is needed. Wolf, 
Winery, and Hagerty (1995) suggest a portfolio comprising three distinct sections which contain 
a variety of information regarding the professional development of the prospective teacher. 
Graves (1992) cautions against standardization of the portfolio to maximize the ownership of the 
portfolio by the student. Since no one had investigated what the constituencies of student 
teaching hold important-, we decided to survey the four groups to determine the perceptions of 
relative importance of some of our program’s requirements. 

Method 

At the conclusion of the Spring 1997 Semester in a college in northwestern New York, 
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portfolio data were collected from secondary education methods students and secondary 
education student teachers, as well as cooperating teachers and teacher educators. Subjects were 
directed to complete a form which required each of the items (portfolio requirements) to be 
ranked from 1 (high) to 15 (low) in terms of perceived utility. 

During a previous pilot study, it was determined that 60 requirements were considered 
by teachers and students to be too large a number of requirements for one semester. The items 
which were the 10 most valuable and the 5 least valuable were compiled to form a list of 15 (see 
Table 1) portfolio submissions. The 5 least valuable items were randomly placed on the list of 
portfolio requirements. This list of portfolio requirements were presented to the 40 methods 
students, 28 secondary student teachers, 11 teachers, and 8 teacher educators, with directions 
to rank each item from 1 (high) to 15 (low). 

Results 

For each of the 15 portfolio requirement items, a mean was calculated for the 4 groups 
of subjects: the secondary methods students, secondary student teachers, cooperating teachers 
and teacher educators (see Table 2). These means were used as the basis for the ranking of the 
portfolio requirements (see Table 3). 

Examination of these rankings reveal several consistent findings. For example, item 
number 4 (see Table 4), "Design a set of 6 lesson plans; include rationale, objective activities, 
assessments, and assignments and use a variety of Gardner intelligences" was ranked 2nd by 3 
groups (methods, student teachers, teacher educators) and 1st by the cooperating teachers. Three 
of the groups (methods students, student teachers, teacher educators) ranked item number 1 
(Design a lesson plan to teach an important concept to heterogeneously grouped 10th graders, 
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one that flows from Vermette’s 1983 model. Include a visual-diagram, chart, picture, etc. -that 
helps clarify one aspect of the lesson as it is taught) as 1st. In keeping with findings from the 
pilot study, the lowest 5 items (rank 15, 14, 13, 12, 11) also received the lowest 5 rankings in 
this study. 

Correlations were calculated between the student teachers and the methods students with 
the result of r = .979. Correlations were also calculated between the cooperating teachers and 
the teachers with a result of r = .786. The correlation between the combined group of methods 
students and student teachers with the combined group of teacher educators and cooperating 
teachers resulted in r = .79. 



Discussion 



Limitations 

The generalizability of this study’s findings are severely limited by several factors. First, 
the sample size is obviously too small for large scale comparisons, yet is of some value to the 
single institution whose program was used. Second, since it involved only one program, the 
study focused on the demands/requirements of that program. However, since there are no similar 
studies in the literature to contradict the findings, this investigation has pointed researchers in 
a direction that should be noted. Future research along these lines would be warmly welcomed 
and invaluable to teacher education reform efforts. 

Observations 

This study found tht methods students were like student teachers were like teachers were 
like teacher educators. In effect, all four groups held similar portfolio submissions 
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(requirements) to be of similar value. This finding is interesting for several noteworthy reasons: 
1) The notion of ownership of the portfolio entries has been of great debate in the teacher 
education literature. Some (Barton & Collins, 1993; Mokharti, Yellin, Bull, and Montgomery, 
1996; and Paulson, Paulson, and Meyer, 1991) enthusiastically demand that the makers should 
control the entries. We found that the owners/makers indeed will pick well, according to the 
standards of the more veteran members of the profession. In effect, they can be trusted to make 
wise choices and recognize what is of value to themselves and others. They can make these 
"acceptable" choices freely and yet still maintain a high level of credibility. 

2) paradoxically, the data also support the notion that teacher educators can be trusted 
to demand the "right" kinds of outcomes from novices, again using the standards set by veteran 
teachers. While this may seem trivial at first blush, further examination may prove interesting. 
Many outside agencies, including the ETS (NTE), Teach for America, National Examiner 
Boards, all alternative certification regulations, and even the Holmes Group, have sent signals 
of deep mistrust of Teacher Education programs. In one way or another, these all suggest that 
TED colleges need heavy outside supervision and regulation; our preliminay findings suggest 
that the (involved) program has the same priorities as veteran teachers, a factor that is never 
discussed by outside control agencies. 

Interestingly, the involved institution is an NCATE-approved program; perhaps further 
research exploring these issues may wish to examine those factors. Moreover, future research 
may wish to examine the perceptions of other "players" in the teacher education debate. Some 
that are suggested include administrators, politicians, Boards of Education, and parents: our 
findings can’t help but make us wonder what other people think of these program 
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outcomes/portfolio submissions. 

To reiterate: teacher educators actually recognize the value of the same set of desired 
outcomes as do veteran teachers. 

3) the previous sentence should be looked at very closely: it suggests that the rank 

ordering of portfolio submissions also includes the ranking of possible program outcomes. In the 
case of the involoved institution, this is a reality. Note that this insitution has (1) made its 
oucomes public, (2) offered public analysis and debate on the wisdon of those outcomes, (3) and, 
by the portfolio pesentation process, directly connected the outcome to the demonstration of 
mastery by performance. These three conditions ARE NEVER POSSIBLE with standardized 
testing processes. Tests, by definition, "sample" a universe of possible knowledges, do not make 
their specific measurable objectives known and requires a "leap of faith” to connect a correct 
answer to a desired teaching skill. The portfolio process at this institution, partially "validated" 
by the findings of this study, have moved a long way toward improving its credibility. 
Admittedly, this third point is an argument favoring portfolios in teacher education, but that 
position is supported by discovering that all the "players" in the certification process seem to 
have the same perceptions of relative usefulness. 

(4) Finally, one must recall that part of the portfolio "debate" is over the purpose of the 
process. Some argue that they show growth, spark reflective thinking, and promote best work 
or provide samples of student work. 

The findings of this study suggest that they primarily have value as assessment tools, 
which is important, because candidates and programs are evaluated by their ability to present 
evidence of mastery of IMPORTANT outcomes. 
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Table 1 

Portfolio Submissions 



Status: Check most accurate descriptor: 

(l)student (4)teaching 3-7 years 

(2)methods student (5)teaching 8 or more years 

(3)teaching 0-2 years (6)teacher educator 

Directions: Here are 15 submissions for a student teacher's portfolio. Please rank order the value of these activities, with 1 as 

the most valuable, and 15 as the least valuable. 

a) Design a lesson plan to teach an important concept to heterogeneously grouped 10th graders, one that flows from 

Vermette’s 1983 model. Include a visual (diagram,chart, picture, etc.) that helps clarify one aspect of the lesson as 
it is taught. 

b) Given a specific topic of content, create acceptable three part behavioral objectives, one for EACH of the six Bloom 

levels. 

c) Compare the depictions of teachers in four Hollywood films with the reality of Siegel and one other teacher. 

d ) Design a set of 6 lesson plans; include rationale, objective, activities, assessments and assignments and use a variety 

of Gardner intelligences. 

e) Write and sing a song describing the ways that technology could be used to enhance specific lesson plans. 

0 Construct 3 interdisciplinary team taught lesson plans that would be acceptable to Madeline Hunter and which would 

be used at the middle school level. 

g) Cite information from three recent articles that is directly applicable to one’s job as a teacher and indicate how that 

information will be used (explain the role of scholarship to a practicing teacher). 

h) Create a lesson plan for demonstration using the model found in Good & Brophy (1997) and which teaches a skill. 

0 Identify 50 key generalizations about teaching found in Good & Brophy and convert them to T/F statements. 

j) Outline an interdisciplinary unit designed for 8th graders. Choose six of the instructional activities included and create 

one anticipatory set for EACH of those six activities. 

k ) Describe the essential components of Cooperative Team Learning, explaining the cognitive and motivational aspects 

involved. 

0 Build and defend a set of student teams for a particular classroom situation. 

— m) Create a lesson plan for an 80 minute lesson taught with cooperative learning structures, one that avoids Vermette’s 

(1994) fatal flaws and which involves the use of a popular teen song. 

n ) Evaluate the multicultural aspects of five lesson plans, offering suggestions in the face of weaknesses. 

— o) Design a theory of homework that utilizes its research base and your experiences. Define the concept "to study” and 

describe examples and non-examples. 
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Table 2 



Means of Portfolio Requirements 



Portfolio 

Requirement 

s 


Teachers 


Teacher 

Educators 


Secondary 

Student 

Teachers 


Secondary 

Methods 

Students 


a 1. 


5.8 


2.5 


1.4 


3.4 


b 2. 


6.9 


7.6 


5.6 


5.5 


c 3. 


13.7 


12.5 


12.8 


13.4 


d 4. 


3.45 


3.1 


1.6 


3.55 


e 5. 


12.45 


13.2 


14.6 


13.7 


f 6. 


5.9 


7.8 


3.0 


4.4 


g 7. 


14.4 


8.3 


9.4 


9.6 


h 8. 


8.4 


7.6 


6.2 


5.55 


i 9. 


12.4 


12.5 


12.4 


10.78 


j io. 


6.7 


6.3 


4.0 


5.21 


k 11. 


4.5 


6.7 


7.6 


7.36 


1 12. 


6.6 


8.5 


9.8 


8.65 


m 13. 


7.45 


4.9 


6.2 


6.01 


n 14. 


4.7 


5.7 


9.6 


8.95 


o 15. 


8.45 


7.75 


12.6 


9.8 




N = 11 


N = 8 


N = 28 


N = 40 ! 
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Table 3 

Rankings of Portfolio Requirements 
(Most Valuable = 1; Least Valuable = 15) 



Portfolio 

Requirements 


Teachers 


Teacher 

Educators 


Secondary 

Student 

Teachers 


Secondary 

Methods 

Students 


a 1. 


4 


1st 


1st 


1st 


b 2. 


8 


7 


5 


5 


c 3. 


14 


12 


14 


14 


d 4. 


1st 


2nd 


2nd 


2nd 


e 5. 


13 


14 


15 


15 


f 6. 


5 


9 


3rd 


3rd 


g 7. 


15 


10 


9 


11 


h 8. 


10 


7 


7 


6 


i 9. 


12 


12 


12 


13 


j 10. 


7 


5 


4 


4 


k 11. 


2nd 


6 


8 


8 


1 12. 


6 


11 


11 


9 


m 13. 


9 


3rd 


6 


7 


n 14. 


3rd 


4 


10 


10 


o 15. 


11 


8 


13 


12 




N = 11 


N = 8 


N = 28 


N = 40 





< 
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Table 4 

Means of Portfolio Requirements 



Portfolio 

Requirements 


Teachers and 

Teacher 

Educators 


Student 
Teachers and 
Methods 
Students 


a 


i. 


4.41 


2.58 


b 


2. 


7.19 


5.54 


c 


3. 


13.19 


13.21 


d 


4. 


3.30 


2.74 


e 


5. 


12.76 


14.07 


f 


6. 


6.70 


3.82 


g 


7. 


11.83 


9.50 


h 


8. 


8.06 


5.82 


i 


9. 


12.44 


11.44 


j 


10. 


6.53 


4.71 


k 


11. 


5.42 


7.46 


1 12. 


7.40 


9.12 


m 


13. 


6.38 


6.09 


n 


14. 


5.12 


9.21 


0 


15. 


8.16 


10.95 




N = 19 


N = 68 
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